The present invention relates to the field of image processing and data distribution. In particular the present invention discloses methods and apparatus for processing, transmitting, and post-processing panoramic image information.
Most cameras only record a view within a small viewing angle. Thus, a typical conventional camera only captures an image in the direction that the camera is aimed. Such conventional cameras force viewers to look only at what the camera operator chooses to focus on.
Panoramic cameras capture a much wider field of field. A panoramic camera constructed using an optical xe2x80x9cfish-eyexe2x80x9d lens may capture at most a 210-degree field of view. A panoramic camera constructed with a catadioptric lens captures a full 360-degree panoramic image. With a large panoramic field of view, individual panoramic image viewers can decide what they will focus on.
Conventional images are often delivered by electronic means. For example, television and the Internet deliver conventional images across wired and wireless electronic media. However, there are no standard means of delivering panoramic images electronically. Since panoramic images are so large, it is difficult to deliver panoramic images using conventional image transmission techniques. To further compound the problem, motion panoramic images traditionally require a very high bandwidth channel for electronic distribution. Thus, it would be desirable to have an electronic distribution system that efficiently distributes motion panoramic image information.
Another difficulty with motion panoramic images is that a user often desires to rapidly pan the view from one area of the panorama to another. Because of the limited bandwidth available for sending the panoramic image from a motion panoramic image server to a client system, such rapid pan operations result in significant latency delays and/or temporal tearing between the time the user performs the pan operation and the display of the resulting view. Thus, it would be desirable to utilize the available bandwidth to provide both high-quality data (for the portion of the motion panoramic image being viewed) and lower-quality data (for the portions of the motion panoramic image that are not being viewed and dependent on the likelihood of that portion being viewed).
Yet another problem with the prior art distribution of motion panoramic images is caused by the latency between a user""s command and the result of the user""s command. This latency is a function of the time required to send the command to the image encoder and the time required by the encoder to propagate the response to the command back to the user. This latency can be of the order of tens of seconds. Thus, it would be advantageous to provide technology that allows the user""s command to be satisfied by a server between the encoder and the client.
Although the communication link between server computers and client computers are beginning to have more bandwidth with the use of digital subscriber lines (DSL) and ISDN, the amount of information flowing across these larger bandwidth links are more complex, are of higher resolution, and higher frame rates. Thus, bandwidth limitations occur even for these more advanced communication links. Thus, it would be advantageous to provide techniques for using the available bandwidth to present a high-quality image while at the same time maintaining sufficient information about the non-presented portions of the panorama to minimize the consequences of panning outside of the high-quality image.
One embodiment discloses an electronic image distribution apparatus for distributing motion panoramic images. The main portion of the apparatus is a motion panoramic image server that has a source of motion panoramic images. The source of panoramic image frames can be a motion panoramic image playback device or a connection to a motion panoramic camera system. The motion panoramic image server transforms the panoramic image frames into an intermediate planar representation that is more conducive for electronic transmission. The motion panoramic image server then transmits the motion panoramic images to client systems.
In addition, one preferred embodiment discloses methods and apparatus for obtaining portions of a panorama over a limited bandwidth link. The invention uses an encoder array to multiply encode the panorama into tiles that have different characteristics. The computer that uses the encoded data to present a view into the panorama subscribes to the tiles containing the encoded data. The tiles that are subscribed-to are selected based on the bandwidth available to transfer the tiles and the quality of the data in the tiles. The computer selects higher-quality tiles near the viewpoint and lower-quality tiles dependent on the distance the tile is from the viewpoint. As the viewpoint is altered (such as by a pan operation) the computer dynamically adjusts the tile subscription to present a quality image. If the pan operation moves the viewpoint such that data from lower-quality tiles is required to present the view, the data is presented using the lower-quality data until the higher-quality data resulting from the new tile subscription arrives. The presentation of the lower-quality data can be reduced or avoided by limiting the range of an allowable pan, by predicting a future need for high-quality tiles based on a pan history, by providing tiered tiles near the viewpoint to reduce temporal and other presentation artifacts.
Another preferred embodiment provides an entire panorama containing distributed-quality data to the client computer. In this embodiment, the panorama is encoded such that a portion of the panorama is encoded for high-quality, other portions of the panorama are encoded for reduced-quality, and yet other portions of the panorama are encoded for low-quality (one skilled in the art will understand that more than three levels of quality can be used within the panorama). Multiple encodings of the panorama differing by the placement of the relative quality portions are provided to the client. The client is thus able to select the panorama encoding that includes a high-quality portion responsive to the client""s viewpoint. When the viewpoint moves away from the high-quality portion of the currently selected panorama encoding, the client subscribes to a different panorama encoding that includes a high-quality portion at responsive to the new viewpoint.
Preferred embodiments include (without limitation) system methods, client computer side methods, server computer side methods, server apparatus, client apparatus, and computer program products for the server computer and the client computer.
Other objects, features and advantages of present invention will be apparent from the company drawings and from the following detailed description.